Video imaging using multi-ping sonar

ABSTRACT

A sonar system comprising a sonar transmitter, a very large array two dimensional sonar receiver, and a beamformer section transmits a series of sonar pings into an ensonified volume of fluid at a rate greater than 5 pings per second, receives sonar signals reflected and scattered from objects in the ensonified volume, and beamforms the reflected signals to provide a video presentation and/or to store the beamformed data for later use. The parameters controlling the sonar system are changed so that the beamformer section treats the data from the receiver section with more than one set of parameters. The stream of data is treated either in parallel or in series by different beamforming methods so that at least one beam from the beamformer has more than one value.

RELATED PATENTS AND APPLICATIONS

The following US patents and US patent applications are related to thepresent application: U.S. Pat. No. 6,438,071 issued to Hansen, et al. onAug. 20, 2002; U.S. Pat. No. 7,466,628 issued to Hansen on Dec. 16,2008; U.S. Pat. No. 7,489,592 issued Feb. 10, 2009 to Hansen; U.S. Pat.No. 8,059,486 issued to Sloss on Nov. 15, 2011; U.S. Pat. No. 7,898,902issued to Sloss on Mar. 1, 2011; U.S. Pat. No. 8,854,920 issued to Slosson Oct. 7, 2014; and U.S. Pat. No. 9,019,795 issued to Sloss on Apr. 28,2015; U.S. patent application Ser. Nos. 14/927,748 and 14/927,730 filedon Oct. 30, 2015, Ser. No. 15/978,386 filed on May 14, 2018, Ser. No.15/908,395 filed on Feb. 28, 2018, Ser. No. 15/953,423 filed on Apr. 14,2018, Ser. No. 16/693,684 filed Nov. 11, 2019, and 62/931,956 and62932734 filed Nov. 7, 2019, Ser. No. 16/362,255 filed on Mar. 22, 2019,and 62/818,682 filed Mar. 14, 2019 and are also related to the presentapplication. The above identified patents and patent applications areassigned to the assignee of the present invention and are incorporatedherein by reference in their entirety including incorporated material.

FIELD OF THE INVENTION

The field of the invention is the field generating and receiving ofsonar pulses and of visualization and/or use of data from sonar signalsscattered from objects immersed in a fluid.

OBJECTS OF THE INVENTION

It is an object of the invention to improve visualization using sonarimaging. It is an object of the invention to measure and record thepositions and orientations, and images of submerged objects. It is anobject of the invention to improve resolution of sonar images. It is anobject of the invention to present sonar video images at increased videorates. It is an object of the invention to rapidly change the sonarimage resolution between at least 2 pings of a series of pings. It isthe object of the invention to change rapidly change the direction ofthe field of view on sonar images between at least 2 pings of a seriesof pings.

SUMMARY OF THE INVENTION

A series of sonar pings are sent into an insonified volume of water andreflected or scattered from submerged object(s) in the insonified volumeof water. One or more large sonar receiver arrays of sonar detectors areused to produce and analyze sonar data to produce 3 dimensional imagesof the submerged object(s) for each ping. One or more parameterscontrolling the sonar imaging system are changed between pings to changethe series of images. The resulting changed images are combined togetherto produce an enhanced video presentation of the submerged objects at anenhanced video frame rate of at least 5 frames per second. More than oneof the parameters used to control the sonar imaging system are used toproduce different 3D images from the same ping in a time less than thetime between two pings.

BRIEF DESCRIPTION OF THE DRAWING

FIG. 1 shows a sketch of the layout where the method of the inventionmay be used.

FIGS. 2A, 2B and 2C show side elevation, plan view and end elevationviews of the sonar transmitter of the invention.

FIG. 3 shows possible configurations of the sonar transmitter of theinvention.

FIGS. 4A and 4B show the sonar transmitter of the invention sending outpings in a 50 degree included angle and a 25 degree included angle.

FIGS. 5A, 5B and 5C show plan view, side elevation, and end elevationviews of the sonar receiver of the invention.

FIG. 6 shows a flow chart of the method of the invention.

DETAILED DESCRIPTION OF THE INVENTION

It has long been known that data presented in visual form is much betterunderstood by humans than data presented in the form of tables, charts,text, etc. However, even data presented visually as bar graphs, linegraphs, maps, or topographic maps requires experience and training tointerpret them. Humans can, however, immediately recognize andunderstand patterns in visual images which would be difficult for eventhe best and fastest computers to pick out. Much effort has thus beenspent in turning data into images.

In particular, images which are generated from data which are notrelated to light are often difficult to produce and often require skillto interpret. One such type of data is sonar data, wherein a sonarsignal pulse is sent out from a sonar generator into a volume of seawater or fresh water of a lake or river, and reflected sound energy fromobjects in the insonified volume is measured by a sonar receiver.

The field of underwater sonar imaging is different from the fields ofmedical ultrasonic imaging and imaging of underground rock formationsbecause there are far fewer sonar reflecting surfaces in the underwaterinsonified volume. Persons skilled in the medical and geological artswould not normally follow the art of sonar imaging of such sparsetargets. FIG. 1 shows a sketch of the system of the invention. A vessel10 carrying the apparatus 11 of the invention is on the surface 14 of abody of water which we will call a part of a sea. The water rests on aseabed 13. It is understood that any fluid that supports sound waves maybe investigated by the methods of the present invention. The apparatus11 generally comprises a sonar ping transmitter (or generator) and asonar receiver, but the sonar transmitter and receiver may be separatedfor special operations. Various sections of the apparatus are eachcontrolled by controllers which determine parameters required foroptimum operation of the entire system. In the present specification, aparameter is a specific value to be used which can be changed rapidlybetween pings. The parameters may be grouped in sets and the set can beswitched, either by hand or automatically according to a criterion. Thedecision to switch parameters may be made by an operator or madeautomatically based on information gained from prior pings sent out bysonar transmitter or by information gained from the current ping. FIG. 1shows the sonar transmitter sending out pulses of sound waves 12 whichpropagate into the water in an approximately cone shaped beam. Thepulses 12 strike objects in the water such as stones 15 on the seabed13, an underwater vessel 17, a swimming diver 18, and a sea wall 16. Thevessel 17 may either be manned or be a remotely operated vessel (ROV).The objects underwater that have a different density than the sea waterreflect pulses 19 as a generally expanding waves back toward theapparatus 11.

The term “insonified volume” is known to one of skill in the art and isdefined herein as being a volume of fluid through which sound waves aredirected. In the present invention, the sonar signal pulse of soundwaves is called and defined herein as a ping, which is sent out from oneor more sonar ping generators or transmitters, each of which insonifiesa roughly conical volume of fluid. A sonar ping generator is controlledby a ping generator controller according to set of ping generatorparameters. Ping generator parameters comprise ping sonar frequency,ping sonar frequency variation during the ping pulse, ping rate, pingpulse length, ping power, ping energy, ping direction with respect aping generator axis, and 2 ping angles which determine a field of viewof the objects. A ping generator preferably has a fixed surface ofmaterial 22 which is part of a sphere, but may shaped differently.Preferred ping generators of the invention are sketched in FIGS. 2through 4. FIG. 2A shows a ping generator cross section 20 with piezoelectric elements 21 sandwiched between electrically conductingmaterials 22 and 23. Material 25 between the piezo electric elements iselectrically insulating. The electrically conducting material 22 ispreferably a solid sheet of material which is grounded and is in contactwith the seawater. Material 22 is thin enough that ultrasonic pressurewaves can easily pass through it, but thick enough that water does notleak through it and get into the interior of the ping generator. Theother end of the piezoelectric material elements 21 is energized byapplying an ultrasonic frequency voltage to electrical elements 24 whichare separated electrically from each other and which energize groups ofpiezo electric elements 21 to vibrate with the same phase and frequency.Wires 25 are sketched to show the electrical connections to thedifferent segments 24. The plan view of the transmitter shows theelements 24 in FIG. 2B shown segmented into 9 segments. FIG. 3 showsother preferred segmentation schemes useful in the method of theinvention. FIG. 4A shows the beam pattern of the outgoing sonar waves ifall the elements 21 are energized with the same phase and frequencyelectrical signal. FIG. 4B shows the beam pattern of the outgoing sonarwaves if only the elements 21 in the center section of FIG. 2B areenergized with the same phase and frequency electrical signal. For therelative size and curvature of the surfaces 25 of FIG. 4A, the full beamhas a divergence of 50 degrees and the restricted beam shown in FIG. 4Bhas a divergence of 25 degrees. By energizing appropriate combinationsof electrodes, the beam may be sent out up, down, left, or right.

Ping generators of the prior art could send out a series of pings with aconstant ping frequency during the ping. Ping frequencies varying intime during the sent out ping are known in the prior art. Changing theping frequency pattern, duration, power, directions, and other pingparameters rapidly and/or automatically between pings in a series hasnot heretofore been proposed. One method of the invention anticipatesthat the system itself uses the results from a prior ping can beanalyzed automatically to determine the system parameters needed for thenext ping, and can send the commands to the various system controllersin time to change the parameters for the next ping. When operating in awide angle mode at a particular angle and range, for example, a newobject anywhere in the field of view can signal the system controllersto send the next outgoing ping the direction of the object, decrease thefield of view around the new object, increase the number of pings persecond according to a criterion based on the distance to the object, setthe ping power to optimize conditions for the range of the object, etc.Most preferably, the system can be set to automatically change any orall system parameters to optimize the system for either anticipated orin reaction to unanticipated changes in the environment.

In a particularly preferred embodiment, the controller system may be setto change the sent out frequency alternately between a higher and alower frequency. The resulting images alternate between a higherresolution and smaller field of view for the higher frequency, and alower resolution and a larger field of view for the lower frequency. Thealternate images may then be stitched after the receiver stage toprovide a video stream at half the frame rate of the system availablewith unchanged parameters, but with higher central resolution and widerfield of view, or at the same frame rate by stitching neighboringimages.

Intelligent steering of the high-resolution, focused field of view on toa specific target of interest would mean that this technology would notnecessarily be limited only to short range applications. If only one ofthe four steered pings, for example, needs to be continuously updated togenerate real-time images, then the range limit could be significantlyextended. The intelligent focusing may be implemented in a mode wherebya low-frequency, low-resolution ping with a large field of view is usedto locate the target of interest. The subsequent high-frequency,high-resolution ping may then be directed to look specifically at theregion of interest without having to physically steer the sonar head.

In this particularly preferred embodiment, additional intelligent andpredictive processing and inter-frame alignment may be used to accountfor and track motion and moving objects. The priority of frameprocessing may be adapted to allow focus and higher refresh rate ofimages including the primary target, for example with the field of viewcentered on a primary target, or moving objects requiring the imagesthat represent a portion of the field of view containing moving objectto be updated more frequently.

The sonar receiver of the invention is a large array of pressuremeasuring elements. The sonar receiver is controlled by a sonar receivercontroller according to set of sonar receiver parameters. The array ispreferably arranged as a planar array shown in FIG. 5 because it issimpler to construct, but may be shaped in any convenient form such as aconcave or convex spherical form for different applications. The arrayhas preferably 24 times 24 sonar detecting elements, or more preferably48 times 48 elements, or even more preferably 64 time 64 detectors, ormost preferably 128 times 128 elements. A square array of elements ispreferred, but the array may be a rectangular array or a hexagonal arrayor any other convenient shape. The detector elements are generallyconstructed by sandwiching a piezo electric material between twoelectrically conducting materials as shown for the sonar transmitter,but with an electrical connection to each element in the array. When areflected sonar ping reaches the sonar detecting element, the element iscompressed and decompressed at the sonar ping frequency, and produces ananovolt analog signal between the electrically conducting materials.The nanovolt signals are amplified and digitally sampled at a sonarreceiver sampling rate controlled by the sonar receiver controller, andthe resulting digital signal is compared to a signal related to sent outping signals to measure the phase and amplitude of the incoming sonarsignals for each receiver element. The amplification or gain for theincoming sonar signals is controlled by the sonar receiver controller.If the sonar ping frequency is changed rapidly between pings, thesampling rate may also be changed to reflect the changed ping frequency.The incoming sonar ping is divided into consecutive slices of time,where the slice time is related to the slice length by the speed ofsound in the water. A slice time parameter is set by the sonar receivercontroller. For example, pings arriving from more distant objects canhave wider slices than pings reflections from closer objects. Each slicecontains a number of sonar wavelengths as the pulse travels through thewater. The sonar receiver preferably has sonar receiver parameterscontrolled by the sonar receiver controller to have, for example,programable phase delays between the detector elements digital samplingtimes may be varied to achieve the same result. The sonar receiver mayhave parameters controlled by the sonar receiver controller which can beset to change the amplification or gain of the nanovolt electricalsignals during the incoming sonar ping reflected signals. Prior art timevarying gain (TVG) systems have used preplanned amplification ramps tocorrect for attenuation in the water column. This gain is applied basedon range (distance from transmitter), but the gain profile does notchange from ping to ping. Generally, the attenuation of the ultrasonicwaves is higher for higher ping frequencies. Prior art changed theamplification factor by a preplanned schedule to even out the signalsbetween the received first slice and the last slice of a ping. Prior TVGdid not allow for the increased absorption by soft mud on the seafloor,for example. Since mud absorbs sound waves, the reflected sound wavesare less intense as soon as the reflected slice reaches the mud. The TVGis changed on the next ping to boost the signals that reflect or arescattered by the mud. In the same way, the TVG is changed to boost orreduce the gain for slices that more strongly reflect or are scatteredby a hard, highly reflecting object like the sea wall shown in FIG. 1.

A phase and amplitude of the pressure wave coming into the sonarreceiver is preferably assigned to each detector element for eachincoming slice, and a phase map may be generated for each incomingslice. A phase map is like a topographical map showing lines of equalphase on the surface of the detector array.

FIG. 5C sketches a reflected ping 54 reflected by first object at arange of 20 detector widths from the detector. The first object is on aline starting from the center of the detector and perpendicular to thedetector surface. The scattered ping is shown having a spherical surfaceto reflect a wave with origin at the surface of the first object. Thephase map for this ping will be a series of circular regions centered inthe center of the detector, all having the same phase, and movingoutward from the center of the detector as the various slices of theping are analyzed. Reflected ping 55 indicates a second object locatedfurther away from the detector than the first object, and at an angle of5 degrees to the right of the center line. Reflected ping 56 shows athird object located yet further away from the detector, and at an angleof 10 degrees to the left of the center line. Pings 55 and 56 producesimilar rings originating to the left and right of the detector, andexpanding as slightly elliptical rings outwardly from their centers(which are not located on the detector for the angles shown).

Applying additional gain control can be incorporated with PhaseFiltering.

Phase map and data cleanup and noise reduction may be done optionally inthe sonar receiver or in a beamformer section. The phase map and/or thedigital stream of data from the detector are passed to the beamformersection, where the data are analyzed to determine the ranges andcharacteristics of the objects in the insonified volume.

The range of the object is determined by the speed of sound in the waterand the time between the outgoing ping and the reflected ping receivedat the receiver. The data are most preferably investigated by using aspherical coordinate system with origin in the center of the detectorarray, a range variable, and two angle variables defined with respect tothe normal to the detector array surface. The beamformer section iscontrolled by a beamformer controller using a set of beamformerparameters. The space that the receiver considers is divided into aseries of volume elements radiating from the detector array and calledbeams. The center of each volume element of a beam has the same twoangular coordinate and each volume element may have the same thicknessas a slice. The beam volume elements may also preferably have thicknessproportional to their range from the detector, or any othercharacteristic parameters as chosen by a beamformer controller. Therange resolution is given by the slice thickness.

The beamformer controller controls the volume of space “seen” by thedetector array and used to collect data. For example, if the sonartransmitter sends out a narrow or a broad beam, or changes the directionof the sent out beam, the beamformer may also change the system to onlylook at the insonified volume. Thus, the system of the inventionpreferably changes two or more of the system parameters between the samepings to improve the results. Some of the parameters controlled by thebeamformer controller are:

-   -   Field-of-view    -   Minimum and maximum beamformed ranges    -   Beam detection mode such as (First Above Threshold FAT or        maximum amplitude (MAX) or many other modes as known in the art)    -   Range resolution    -   Minimum signal level included in image    -   Image dynamic range    -   Array weighting function (used to modify the beamforming        profile)    -   Applying additional gain post beamforming (this can be        incorporated with Thresholding).

The incoming digital data stream from each sonar detector of thereceiver array has typically been multiplied by a TVG function. Atriangular data function ensures that the edges of the slices havelittle intensity to reduce digital noise in the signal. The TVG signalis set to zero to remove data that is collected from too near too and tofar away from the detector, and to increase or decrease the signaldepending on the situation.

In the prior art, the data have been filtered according to a criterion,and just one volume element for each beam was selected to have a value.For example, if the data was treated to accept the first signal in abeam arriving at the detector having an amplitude above a definedthreshold (FAT), the three dimension point cloud used to generate animage for the ping would be much different from a point cloud generatedby picking a value generated by using the maximum signal (MAX). In theFAT case, the image would be, for example, of fish swimming through theinsonified volume and the image in the MAX case would be the image ofthe sea bottom. In the prior art, only one range in each beam would showat most one value or point and all the other ranges of a single beamwould be assigned a zero.

In the present invention, the data stream is analyzed by completing twoor more beamformer processing procedures in the time between two pings,either in parallel or in series. In a video presentation, the prior artshowed a time series of 3D images to introduce another, fourth dimensiontime into the presentation of data. By introducing values into more thanone volume element per ping, we introduce a 5^(th) dimension to thepresentation. We can “see” behind objects, for example and “through”objects and “around” objects to get much more information. We can usevarious data treatments to improve the video image stream. In the sameway, other ways of analyzing the data stream can be used to accomplishprovide cleaner images, higher resolution images, expanded range images,etc. These different images imaging tasks to can be used on only oneping. The different images may be combined into a single image in avideo presentation, or in more than one video at the frame rate the sameas the ping rate.

If we are surveying a seawall, we Beamform the data before the wall (seabottom—oblique to beams (low backscatter) soft (low intensity signalsreturned)) differently from the harbour wall (orthogonal to beams (highback scatter) hard, high intensity. If we know where a seawall is from achart, the beamformer can use GPS or camera data to work out what rangesare before the wall and what are after and change TVG in the middle ofthe returned ping.

If we know the sea depth we can specify two planes, SeaSurfacePlane andSeatBottomPlane only data between the planes will be processed and sentfrom the head to the top end.

A large amount of data generated per second by prior art sonar systemshas traditionally been discarded because of data transmission and/orstorage limits. The present invention allows a higher percentage of theoriginal data generated to be stored for later analysis.

FIG. 6 shows a flowchart of the method of the invention. The start 60 ofthe process of sending out a ping is to set all system parameters forall system controllers. Either all parameters are the same as the lastping, or they have been changed automatically by signals from stages ofthe previous ping. Step 60 sends signal to step 61 to send commands totransmitter 62. Transmitter sends data to receiver controller 63 to setparameters for receiver 64 and start receiver 64. Receiver receivesanalogue signals, samples the voltages from each element, and transmitsdata to the beamformer controller which sends data and instructions tothe Beamformer section.

The beamformer analyses data and decides whether the next ping shouldchange settings, and if so sends signals to the appropriate controllerto change the settings for the next ping. The beamformer analyses thedata in step 67 and decides either on the basis of incoming ping data oron previous instructions whether to perform single or multiple types ofanalysis of the incoming ping data. For example, the beamformer couldanalyze the data using both the FAT and MAX analysis, and present bothimages either separately or combined, so that there will be some beamshaving more than one value per beam. The reduced data is sent from step67 to step 68 which stores or sends raw data or image data for furtherprocessing into a video presentation at a rate greater than 5 frames persecond.

Obviously, many modifications and variations of the present inventionare possible in light of the above teachings. It is therefore to beunderstood that, within the scope of the appended claims, the inventionmay be practiced otherwise than as specifically described.

1. A method of recording a 3D sonar image, comprising; a) transmitting aseries of sonar pings into a first volume of water, the series of sonarpings transmitted from a sonar ping transmitting device at a rate atleast 5 pings per second, wherein the sonar ping transmitting device iscontrolled by sonar ping transmitting parameters, and wherein each sonarping transmitting parameter is chosen from predetermined list of sonartransmitting parameter settings; b) receiving sonar signals reflected orscattered from objects in the first volume of water from each of theseries of sonar pings, the received sonar signals received by a largetwo dimensional array sonar receiving device; c) wherein the sonarreceiving device is controlled by sonar receiving parameters, and; d)beamforming the received sonar signals from each of the series of sonarpings with a sonar beamforming device to form a three dimensional (3D)sonar image of the objects reflecting or scattering the received sonarsignals, wherein the sonar beamforming device is controlled by a set ofsonar beamforming parameters, and wherein each sonar beamforming deviceparameter is chosen from predetermined list of sonar beamforming deviceparameter settings; e) changing at least one beamforming parameter inthe time between any two sonar pings of the series of sonar pings toproduce at least two real time beamformed data sets for the same ping;and f) combining the at least two real time beamformed data sets toproduce a single video frame image in the time between two sonar pings.2. The method of claim 1, where the combined beamformed data sets havemore than one value for at least one beam.
 3. The method of claim 2,where the two real time data sets are the FAT data set and the MAX dataset.
 4. The method of claim 2, where the two real time data sets havedifferent range settings.
 5. The method of claim 2, where the two realtime data sets have different time varying gain (TVG) settings.
 6. Themethod of claim 2, where the two real time data sets have differentField-of-View (FOV) settings.
 7. A method of real time three dimensional(3D) sonar imaging, comprising: a) insonifying a volume of fluid with aseries of sonar pings, the sonar pings, wherein the series of sonarpings are produced at a rate greater than 5 pings a second; b) receivingfor each of the series of sonar pings sonar signals reflected from oneor more objects in the volume of fluid, wherein the sonar signals arereceived with a large 2D array of sonar signal detectors; c) beamformingthe reflected sonar signals to provide a series of (3D) sonar images ofthe one or more objects; wherein the beamforming procedure is changedfrom ping to ping to produce a series of images having different fieldof view (FOV) for each of at least two consecutive pings.
 8. The methodof claim 7, further comprising; d) stitching at least two consecutiveimages of the series of 3D images produced in step c) to make acomposite 3D image having a wider field of view than any one of theseries of (3D) sonar images of step c).
 9. The method of claim 8,wherein four consecutive images of the series of 3D images havingdifferent fields of view are stitched together to produce a singleimage.
 10. The method of claim 9, further comprising; e) recording avideo of the composite images of step d), wherein the video showscomposite single images stitched from four consecutive images, thecomposite images shown sequentially at a rate greater than 5 images persecond.
 11. The method of claim 7, further comprising; d) identifying anobject of interest from at least one image of the series of 3D images ofthe one or more objects; and e) changing the field of view of at leastone succeeding ping is to provide further images of the object ofinterest.
 12. The method of claim 11, wherein step e) changes the fieldof view so that the beamformed image of the object of interest isapproximately in the center of the changed field of view in succeedingpings.
 13. The method of claim 7, wherein step c) changing thebeamforming procedure from ping to ping includes inserting aprogrammable set of delays in sonar signals received by each element ofthe large array of sonar signal detectors.
 14. The method of claim 8,wherein a subset of the at least two consecutive images of the series of3D images is updated continuously to generate real time images.
 15. Themethod of claim 8, wherein intelligent processing is used to account forand/or track motion of moving objects.
 16. The method of claim 8,wherein predictive processing is used to account for and/or track motionof moving objects.
 17. The method of claim 8, wherein interframealignment is used to account for and/or track motion of moving objects.18. The method of claims 16-18, wherein the portion of the field of viewcontaining the motion of the moving objects is updated more frequentlythan the remaining portions of the field of view.